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ABSTRACT 



The National Center for Education Statistics (NCES) sponsors 
the Teacher Follow-up Survey (TFS) , conducted by the U.S. Census Bureau. The 
TFS provides information about teacher attrition. The sample is selected from 
teachers who responded to the Schools and Staffing Survey in the preceding 
year. The Census Bureau conducted the TFS in 1988-89, 1991-92, and 1994-95. 
During each of these surveys, the Census Bureau reinterviewed a subsample of 
the respondent teachers. In 1994-95, the reinterviews focused on current 
teachers who had remained in teaching after participating in the SASS. The 
TFS reinterview measured response variance in a subset of questions from the 
survey. Seventy-eight percent of the questions evaluated displayed high 
response variance, and only 5% displayed low response variance. The most 
problematic questions involved allocation of teacher time. Recommendations 
for survey improvement center on redesign of the questions about teachers ’ 
main activity and the substitution of "yes/no" questions for "mark all that 
apply." Five attachments contain recommendations, response variance formulas 
and measures tables, a conversion table, and the reinterview questionnaire. 
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I. 



Summary 



The National Center for Education Statistics (NCES) sponsors the Teacher Follow-up 
Survey (TFS), conducted by the U.S. Census Bureau. The TFS provides information 
about teacher attrition. The Census Bureau selects the TFS sample from the teachers who 
responded to the Schools and Staffing Surveys (SASS) in the previous year. The Census 
Bureau conducted the TFS in 1988-89, 1991-92, and 1994-95. 

During each of the three TFS surveys, the Census Bureau reinterviewed a subsample of 
the respondent teachers. The 1994-95 TFS reinterview focused on current teachers, who 
remained in teaching after they participated in the 1993-94 SASS. The 1988-89 and 
1991-92 TFS reinterviews [1], [2], [3] also included both current and former teachers. 

The TFS reinterview measured response variance in a subset of questions from the 
survey: 



• questions the NCES deemed critical, 

• questions suspected to be problematic, and 

• new questions not previously evaluated 

The reinterview evaluated many questions added to the “teaching methods” section of the 
1994-95 TFS. 

High response variance indicates a question is very problematic. Moderate response 
variance indicates a question is somewhat problematic. Questions that are not written 
clearly or that address difficult-to-measure concepts often suffer from moderate or high 
response variance. 

A. Major Findings 

Seventy-eight percent of the questions evaluated displayed high response 
variance. Only five percent displayed low response variance. 

Two questions were particularly problematic: 

la Is your time EQUALLY DIVIDED between two of the above listed 
activities? 

2 What code from the list above best describes the activity you spend most of 
your time on during the work week; that is, what is your main activity? 

These questions were revised after the 1991-92 TFS, but the new layout seems 
difficult for respondents to follow. Besides having high response variance, 44 
percent of the teachers answering "Yes” to question la failed to skip question 2 as 
instructed. 




2 



In the “Main Activity” section of the questionnaire, only question 7a, “What is 
your MAIN teaching assignment at this school, that is, the field in which you 
teach the most classes?” displayed low response variance. The questions 
regarding teacher assignments and certificates (questions 7a through 9b) displayed 
the same problems reported in the 1993-94 SASS reinterview [4], These 
questions used the same format as the 1993-94 SASS, jumping back and forth 
between MAIN and OTHER fields. The jumping between MAIN and OTHER 
seems to have confused the respondents. 

This reinterview again confirms that “mark all that apply” questions tend to be 
problematic. Of the eight categories evaluated in question 33, “Which of the 
following describe your DESIGNATED CLASS?” six displayed high response 
variance. The other two had moderate response variance. Unclearly defined 
categories in this question may have exacerbated the usual problems of “mark all” 
questions. 

Finally, all but one of the 54 questions on teaching methods had moderate or high 
response variance. 

B. Recommendations 



We suggest a reconsideration of the objectives for the “main activity” questions 1- 
3, then redesign and test them. This battery of questions would benefit from the 
expertise of our cognitive researchers. At a minimum, we suggest defining “work 
week” more precisely to help focus respondents on the sponsor’s objective. 

We again recommend (as we did in the SASS 1993-94 report) implementing the 
improvements suggested in Part 1 of the 1992 Teacher Follow-up Survey 
Reinterview Report [2]. In brief, group the questions for the MAIN teaching 
assignment together, then ask the questions for the OTHER teaching assignment. 

In general, we strongly recommend substituting separate “Yes/No” questions for 
each response category of “mark all that apply” questions. Previous work [5], [6] 
has shown that the “mark all” format leads to questionable data. 

Attachment A contains Andrew Zukerberg’s, Demographic Surveys Division 
(DSD), comments and recommendations based on his cognitive research 
experience. We recommend continuing cognitive research into problematic 
questions. Revised questions should be reinterviewed again to determine if 
reliability has improved. 
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II. Methodology 

A. Reinterview Procedures 

In January 1995, the Census Bureau mailed TFS questionnaires to the selected 
sample of current and former teachers. In February, the Census Bureau mailed a 
second questionnaire to each sample person who had not returned the first 
questionnaire. In late March, Census Bureau field representatives (FRs), began 
calling sample persons who had not returned a mail questionnaire. This telephone 
follow-up of nonrespondents ended in mid-June 1995. 

The 1995 TFS reinterview included only current teachers and replicated the 
original interview’s mode. If the original interview was completed by mail, the 
reinterview was completed by mail. If the original interview was completed by 
telephone, the reinterview was completed by telephone. 

At least once a week the Data Preparation Division (DPD) in Jeffersonville, 
Indiana received a list of completed original mail questionnaires. Within a week 
of receiving the list, DPD mailed out the reinterview questionnaires. Accounting 
for the time needed for questionnaires to travel back and forth by mail, we 
estimate that most mail respondents received reinterview questionnaires within 
three to four weeks after completing the original interview. If the respondent did 
not return the reinterview questionnaire within three weeks of the first reinterview 
mailing, DPD mailed a second reinterview questionnaire. 

The telephone reinterviews were done on a flow basis, using paper-and-pencil 
(PAPI) reinterview questionnaires. Telephone reinterview began in early April, 
1995 and ended in early July. As FRs completed and mailed original telephone 
interviews to the Regional Offices (ROs), the ROs prepared the reinterview 
questionnaires and mailed them to the senior field representatives (SFRs) to 
conduct the telephone reinterviews. The SFRs were instructed to call back when 
the teacher was unavailable. 

B. Reinterview Sample Design and Response Rates 

After excluding teachers who were in the 1994 SASS teacher reinterview 
sample, we selected a systematic random subsample of current teachers from the 
TFS sample. Current teacher status was determined by having the schools 
indicate whether those teachers who participated in the 1993-94 SASS were still 
teaching at the school in a teaching or nonteaching capacity, or had left the 
school to teach elsewhere or for a non-teaching occupation [7]. The reinterview 
sample was selected at the same time as the original TFS sample. We selected 
enough extra sample to account for original sample cases that were 
noninterviews or out of scope and for reinterview noninterviews. 
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We aimed to obtain 1,000 completed reinterviews. We completed 870 
reinterviews - 495 mail cases and 375 telephone cases. 

Table 1 shows the reinterview sample sizes and response rates. The overall 
reinterview response rate was 62.7 percent, somewhat disappointing. This low 
response rate was due mainly to the 54.2 percent reinterview response rate for 
telephone cases. The 71.2 percent reinterview response rate for mail cases was 
particularly good for a reinterview. We recommend conducting the next PAPI 
telephone reinterview, for TFS or SASS, from the Telephone Centers (Tcs). 

We generally get higher response rates from the TCs than from field 
reinterviews. We recommend keeping the TC reinterviews PAPI to replicate the 
original interview mode as closely as possible. 

Table 1. 1994-95 Teacher Follow-up Reinterview 
Sample Sizes and Response Rates 



Cases 


Total 


Mail 


Telephone 


Selected for RI 


1545 


715 


830 


Noninterview 
in Original 


96 


9 


87 


Out-of-scope 


62 


11 


51 


Eligible for RI 


1387 


695 


692 


RI Completed 


870 


495 


375 


RI Response Rate 


62.7% 


71.2% 


54.2% 



C. Reinterview Model Assumptions 

The reinterview response error model assumes the reinterview is an independent 
replication of the original interview. 

Independence means that response errors are not correlated between the original 
interview and the reinterview. If the respondents remembered their original 
answers and consciously repeated them in the reinterview, the independence 
assumption would be violated. Lack of independence generally results in 
underestimates of response variance. 

Replication means that the reinterview was conducted under the same 
conditions as the original interview. If the reinterview replicates the original 
interview, the distributions of original and reinterview responses will be the 
same. With quantitative data, neither the means nor the variances of the original 
responses will differ significantly from the reinterview responses. With 
categorical data, the difference between the original proportion in-category and 
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the reinterview proportion in-category, the net difference rate (NDR), will not 
differ significantly from zero. 

Measures Used to Estimate Response Variance 

Random errors of measurement in the survey process (nonsampling error) increase 
the mean square error of the data collected. When the errors are not correlated 
with the answers or with each other, we call this variability “simple response 
variance.” Simple response variance is the average variance of responses to an 
item over repeated applications of the measurement process. 

The index of inconsistency and the gross difference rate (GDR) are the 
principal measures of response variance in categorical data. We estimate an index 
and a GDR for each response category of a question. 

Overall estimates of the index and the GDR for a question, the aggregate index 
and the aggregate GDR, apply to questions with three or more answer categories. 

We use Pearson’s correlation coefficient to measure data reliability for 
quantitative variables. When the means and variances for the two interviews are 
exactly the same, the index equals one minus the correlation. When the response 
variance model assumptions are met, the index is approximated by one minus the 
correlation between the original and reinterview responses (I = 1 - p). In this 
report, we use this approximation for the index. 

This report provides 90 percent confidence intervals for these measures. See 
Attachment B for the formulas used to calculate the measures and the confidence 
intervals. 

1. Index of Inconsistency 

The index of inconsistency estimates the ratio of response variance to total 
variance for an answer category. It is a relative measure of response 
variance. 

The aggregate index is similar to the index of inconsistency but applies to 
the entire question rather than a specific answer category of the question. 

It is an average index of inconsistency across all categories in the question, 
with each category weighted by its relative size. In 2 x 2 tables the index 
of inconsistency and the aggregate index are equal. 

Use this rule of thumb to interpret the index of inconsistency and the 
aggregate index. 
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Index Value 


Response Variance 


Respondent 

Interpretation 


Less than 20 


Low 


Usually not a major 






problem 


Between 20 and 50 


Moderate 


Somewhat 






problematic 


Greater than 50 


High 


Very problematic 



Any of these factors may cause high response variance: 

• The methods used to collect the data need improvement. 

• The question may be unclearly written. 

• The concept itself may not be measurable or difficult to measure. 

• Respondents may not be able to provide reliable information to the 

level of detail asked. 

2. Gross Difference Rate 



The gross difference rate (GDR) is the percentage of responses that 
change into or out of a category between the original interview and the 
reinterview. For a single category, one-half the GDR equals the simple 
response variance. 

The aggregate GDR is the percentage of responses that change between 
the original interview and the reinterview. It applies to the entire 
question, rather than to a specific answer category of the question. 

The GDR is more difficult to interpret than the index of inconsistency. 
Large GDRs indicate serious response variance in the data. 
Unfortunately, a small GDR is no guarantee of good consistency. In a 
low-frequency category, even a small GDR can indicate a serious 
response variance problem. 

3. Net Difference Rate 



With categorical data, the net difference rate (NDR) helps indicate how 
well the reinterview meets the model assumptions. A statistically 
significant NDR suggests that the reinterview may not replicate the 
original survey conditions as well as desired and is an indication that the 
indices of inconsistency are inaccurate. 





With quantitative data, a comparison between the original interview and 
reinterview means and variances provides information analogous to the 
NDR. 

Cross-T abulations 



For a “yes/no” question, the cross-tabulation looks like this: 



Reinterview 

Response 


Original Response 


Total 


N/A 


Subtotal 


Yes 


No 


Total 












N/A 












Subtotal 






n 


a -H c 


b-Hd 


Yes 






a -H b 


a 


b 


No 






c -H d 


c 


d 



where 

n = the number of respondents who answered the question in both the 
original and the reinterview. 

a = the number of respondents who answered “yes” both times, 

b = the number of respondents whose answer changed from “no” in 

the original to a “yes” in the reinterview, 
c = the number of respondents whose answer changed from “yes” in 
the original to a “no” in the reinterview, 
d = the number of respondents who answered “no” both times. 

We used only cases that respondents answered the question in both the 
original interview and reinterview to compute the response variance 
measures. 

In multi-category questions, these cross-tabulations show the movement 
among answer categories between the original interview and the 
reinterview. Patterns in this movement can provide clues to the reasons 
for inconsistent reporting. In some cases, such movement may even 
suggest question revisions to reduce response variance. 
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In this report, the Census Bureau does not compute the index for answer 
categories with small cell sizes. We used the following rule to determine 
adequate cell size [8], 

2a + b + c ^ 40 and 2d + b + c ^ 40 

In reviewing the documentation, we have determined that this “40 Rule” 
was developed to eliminate manual computations when the index has a 
very wide confidence interval. Current computing resources make this 
restriction irrelevant and we plan to drop it from future analyses. 

Table 2 lists the questions and the response categories in the “mark all 
that apply” questions that did not meet this criterion. 

Table 2. Questions with Too Few Cases to Analyze 



Section 


Questions with Too Few Cases 


Education Field Questions 


lb (01-04 and 06-07) 


Teaching Assignment/ 


9b 


Certification Questions 




Teaching Methods Questions 


33 (7,8, and 11) 



E. Limitations 



We computed response variance measures using unweighted counts and 
estimated the confidence intervals assuming simple random sampling. These 
estimates, therefore, do not account for the complex sample design of the TFS 
and may not perfectly reflect the incidence of response error in the target 
population of current teachers. 

We computed response variance measures using unedited interview and 
reinterview data. We use unedited data so that we can uncover problems 
respondents are having with individual questions, the design of the 
questionnaire, or both. However, by using unedited data our tabulations include 
cases where the respondent answered the question in both the interview and the 
reinterview, even though the respondent should have skipped the question in one 
or both interviews. Edited survey data eliminates responses to questions that 
should have been skipped. 
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The TFS reinterview may not have been independent of the original interviews, 
to the extent respondents remembered and repeated their answers from the 
original interview. 

The reinterviews may not always have replicated the original interviews. The 
proportion of questions with statistically significant NDRs are higher than the 10 
percent we would expect by chance. Specifically, 29 percent of the 132 response 
categories and questions evaluated in the TFS reinterview displayed statistically 
significant NDRs. 

The overall reinterview response rate of 62.7 percent, especially the 54.2 
response rate for telephone cases, may have had a significant impact on the 
accuracy of the reinterview results. 

III. Detailed Results 

We analyzed the responses from 870 teachers on 65 questions. For “mark all that 
apply” questions, each response category is treated as a separate question. 

We did not compute response variance measures for telephone and mail cases 
separately. The reinterview sample was not a controlled sample and was not designed 
to measure mode effects. 

Table 3 summarizes these response variance results. 

Table 3. Response Variance Summary for the 1994-95 TFS 



Level of Response Variance 


Number 


Low 


3 (5%) 


Moderate 


11 (17%) 


High 


51 (78%) 


Total Evaluated 


65 (100%) 


Too few cases to estimate index 


10 


Total 


75 



Attachment C contains tables listing the response variance measures for all the 
questions in the 1994-1995 TFS reinterview. 
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For ease of presentation, we divided the questions into groups based on content: 

• Education Fields Questions (la, lb, 2, 3a, and 3b) 

• Teaching Assignment / Certification Questions ( 7a, 7b, 8a, 8b, 9a, and 9b) 

• Teaching Methods Questions (3 1 , 33, 34, 36, 39, 4 1 , 43, 44, and 49) 

In each group, we discuss only the questions that exhibited moderate response variance 
(indices between 20 and 50) or high response variance (indices greater than 50). 
Estimates of reliability are given with 90 percent confidence intervals in parentheses. 

A. Education Field Questions 



We evaluated questions la, lb, 2, 3a, and 3b regarding education fields. We 
tabulated the response categories (“education fields”) of question lb as seven 
separate “mentioned/not mentioned” questions. This series of questions tries to 
determine the “work activities” of teachers. For the five questions with enough 
data to estimate the index, four had moderate or high response variance. 

A complete redesign of this series of questions is in order. In the 1992 TFS this 
series of questions was asked as 1 question with 2 categories, “Main activity” 
and “Other main activity,” with a check box to mark if the teacher’s time was 
equally divided between the two. The revisions made after the 1992 TFS did not 
work. Moderate to high response variance is not the only problem. These 
questions exhibit frequent skip pattern errors, which suggest they need revision, 
regardless of their reliability. We also recommend cognitive testing of the term 
“work week” to make sure it elicits the type of responses the sponsor desires. 

Question la is very problematic. It has high response variance with an index of 
67.0 (60.3, 74.7). The GDR also suggests this question is problematic; 22.0 
percent (19.7%, 24.4%) of the respondents changed their answer between the 
original interview and the reinterview. 
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EDUCATION FIELDS 

(Use codes to answer items la, lb, 2, 3a and 3b) 

01 Teaching in an elementary or secondary school 

02 Working in an elementary or secondary school with an assignment OTHER THAN 
teaching 

03 Working in an occupation outside of elementary or secondary education 

04 Attending a college or university 

05 Caring for a family member 

06 Retired 

07 Other 

Q la: Is your time EQUALLY DIVIDED between two of the above activities ? 

1 0 Yes 
lONo 



We noted these deficiencies, which may have contributed to this question’s high 

response variance. 

• The title, “EDUCATION FIELDS,” does not accurately describe most of 
the activities on the list. 

• “Time” is not defined and the respondents may have included their 
nonwork activities outside the normal “work week.” Respondents may 
also use inconsistent interpretations of “time” between the two 
interviews. 

• Forty-four percent of the respondents who answered “Yes” failed to skip 
question 2 as instructed, suggesting that many respondents were confused 
by the term “EQUALLY DIVIDED.” 
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Q lb: What are the codes from the list above? 

□ □ and □□ 

Because of the inconsistent reporting to question la, only 83 respondents filled 
both boxes of question lb in both interviews. Only one activity, “Caring for 
family members,” provided enough cases to estimate the index. This activity 
displayed low response variance with an index of 17.9 (9.7, 33.0). 

Of the 266 respondents who answered this question in the original interview, 
one-fourth reported only one activity. And 75 percent of these one-activity 
respondents reported that activity again in question 2, when they should have 
skipped it. 

Q 2: What code from the list above best describes the activity you spend 

most of your time on during the work week; that is, what is your main 
activity? 

This question displayed high response variance, with an aggregate index of 
inconsistency of 62.3 (46.4, 83.7). Forty-three percent of the respondents failed 
to follow the skip pattern and answered this question when they were instructed 
to skip it. 

Q 3a: Do you spend time on any other activity from the list above? 

OYes 

ONo 

This question displayed high response variance with an index of 66.8 
(60.9, 73.5). The GDR was very problematic; 32.2 percent (29.2%, 35.3%) of 
the respondents changed their answer between the original interview and 
reinterview. 

Once again “time” is not defined. Respondents may interpret the question in 
terms of the “normal” work week or in terms of seven 24-hour days. 

Q 3b: What code from the list above best describes this other activity? 

This question displayed moderate response variance, with an index of 39.4 
(32.6, 48.4) and 29.7 percent (23.8%, 35.7%) of the respondents changing 
answers. 
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B. Teaching Assignment / Certification Questions 

We evaluated questions 7a, 7b(l), 7b(2), 8a, 8b, 9a and 9b regarding teaching 
assignments and certificates. For the six questions that had enough data to 
estimate the index, five had moderate or high response variance. 

These questions displayed the same problems reported in the 1993-94 SASS 
reinterview [4]. 

Questions 7b - 9a: Question 7bl asks respondents if they teach classes in 
OTHER fields besides their MAIN field. Question 9a asks if they have a 
teaching certificate in their OTHER field. About one-third of the 834 teachers 
who answered “No” to 7b 1 failed to mark “Not applicable” to 9a. 

We think, as in the SASS, that respondents have trouble with the OTHER 
teaching assignment questions, 9a and 7b, because the intervening questions (8a, 
b, c) address MAIN teaching assignments. 

We recommend (as in the 1993-1994 SASS reinterview report) implementing 
the specific improvements suggested in Part 1 of the 1992 Teacher Follow-up 
Survey Reinterview Report [2]. 

• Group the MAIN teaching assignments questions (7a, 8a, b, and c) 
together. 

• Group the OTHER teaching assignments questions (7b, 9a, b, and c) 
together. 

Grouping will eliminate the need for respondents to shift their thinking between 
their MAIN and OTHER teaching assignments. 

Here are the results for the individual teaching assignments and certificates 
questions. 

Q 7a: What is your MAIN teaching assignment at this school, that is, the 
field in which you teach the most classes? 

Enter your main teaching assignment field and the two-digit code from 
the list above. If your teaching schedule is divided equally between two 
fields, record either field as your main assignment for this item, mark 
box 1, and enter the code for the other field in question 7b. 
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This question displayed low response variance with an index of inconsistency of 
9.5 (7.7, 11.9). 

Q 7b 1: Do you teach classes in OTHER fields at this school? 

ID Yes 
2DNo 

This question is somewhat problematic. It has moderate response variance with 
an index of 3 1 .9 (26.6, 38.2). About one-tenth (8.2%, 1 1 .6%) of the respondents 
changed their answers between the original interview and the reinterview. 

Q 7b2: In what field do you teach the second most classes? 

Use the assignment field codes listed above. 

This question displayed moderate response variance, with an index of 38.2 
(28.4, 52.7) and 17.7 percent (12.1%, 23.4%) of the respondents changing 
answers. 

Q 8a: Do you have a teaching certificate in this state in your MAIN teaching 
assignment field? 

1 D Yes 
2DNo 

This question displayed moderate response variance with an index of 26.3 
(20.6, 33.6). 
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Q 8b: What type of Certificate do you hold in this field? 

Mark (X) only one box 

1 D Advanced professional certificate 

2D Regular or standard state certificate 

3D The certificate offered in your state to persons who HAVE 

COMPLETED what the state calls an “alternative certification 
program ” 

4D Provisional or other type given to persons who are still 
participating in what the state calls an “alternative 
certification program” 

5D Probationary certificate (the initial certificate issued after 
satisfying all requirements except the completion of a 
probationary period) 

6D Temporary certificate ( requires some additional college 
coursework and/or student teaching before regular 
certification can be obtained) 

7D Emergency certificate or waiver (issued to persons with 

insufficient teacher preparation who must complete a regular 
certification program in order to continue teaching) 



This question displayed moderate response variance with an aggregate index of 
45.7 (40.2, 52.2). The GDR also suggests this question was problematic; 17.9 
percent (15.5%, 20.2%) of the respondents changed their answers between the 
original interview and the reinterview. 

To determine if the moderate response variance was caused by too fine a 
distinction in type of certificate, we combined the first three categories into one 
category (Hold regular certificate or above), the next three categories into 
another category (Still needs to complete requirements), and left the last category 
(Emergency certificate or waiver) by itself. Combining categories still resulted 
in moderate response variance, with an aggregate index of 42.6 (33.0, 54.8) and 
a GDR of 5.8 percent (4.4%, 7.3%). All the inconsistent cases, except one 
changed between the first two combined categories. 
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Q 9a: Do you have a teaching certificate in your OTHER teaching 
assignment field at this school? 

On Not applicable; I do not have a second teaching assignment 
field 
!□ Yes 
20 No 

This question displayed high response variance with an index of 65.7 
(61.2, 70.9). The GDR was very problematic; 37.3 percent (34.5%, 40.0%) of 
the respondents changed their answers between the original interview and the 
reinterview. All three answer categories displayed high response variance. 

After combining the “No” and “Not Applicable” categories, the index remained 
high, 58.9 (51.3, 68.0), with a GDR of 13.9 percent (12.1%, 16.0%). 

C. Teaching Methods Questions 



We evaluated questions 31, 33, 34, 36a-k, 39a-c, 41a-j, 43, 44a-f, and 49a-i 
regarding teaching methods in the teacher’s designated class. We tabulated each 
of the 1 1 “mark all that apply” response categories in question 33 as 1 1 separate 
“mention/not mentioned” questions. We also analyzed each of the 5 quantitative 
categories in question 34 separately. We could estimate the index for 54 of these 
questions and subquestions and 53 displayed moderate or high response 
variance. 

Q 31: What was the subfect matter of your DESIGNATED CLASS last 
semester or grading period? 

Record the two digit code from the list above and the field names 

This question displayed low response variance, with an index of inconsistency of 
12.3(10.1, 14.9). 
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Q 33: Which of the following describe your DESIGNATED CLASS? 
Mark (X) all that apply. 



O Heterogeneous 
o Homogeneous 
o Remedial 
o Special education 
o Gifted 

o Academic/college preparatory 



o Advanced placement/college 
credit 

o Honors Course 
o Vocational 
o Bilingual 
o None of the above 



This question proved problematic. Six of the eight response categories we 
evaluated displayed high response variance. Table 4 shows the index of 
inconsistency, the GDR and the NDR for each response category. 

One cause of this question’s high response variance is its “mark all” format. 
Although “mark all that apply” questions reduce respondent burden, we 
recommend not using this type of question. Research has shown that using 
separate “yes / no” questions results in lower response variance than using a 
“mark all that apply” question [5]. Beyond the “mark all” format, the terms 
“heterogeneous” and “homogeneous” are not clearly defined — heterogeneous 
with respect to what? Race? Income? Academic ability? About four percent of 
respondents answered both “heterogeneous” and “homogeneous” in the original 
survey. 

To analyze this question, we treated each response category as a separate “yes / 
no” question. For example, consider the first response category. We analyzed it 
as an individual question “Would you describe your DESIGNATED CLASS as 
Heterogeneous?” If the respondent marked “Heterogeneous,” we counted that 
case as a “yes.” If the respondent didn’t mark “Heterogeneous,” we counted that 
case as a “no.” If the respondent left the question totally blank, not even 
marking “None of the above,” then we counted that case as item nonresponse. 

Of the 870 completed reinterview cases, this question was left totally blank in 
the original interview, but not in the reinterview, 1 1 times, and 10 times in the 
reinterview, but not in the original interview. 



Finally, six of the eight response categories evaluated had statistically significant 
NDRs, suggesting that the response error model assumptions may not hold for 
this question. 




25 



Table 4. 

Q 33: Which of the following describe your Designated CLASS? 
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Response category 


Index 


GDR 


NDR 


Heterogeneous 


57.3 

(51.8, 63.5) 


23.2 

(20.8, 25.6) 


-5.1* 

(-2.3, -7.8) 


Homogeneous 


69.5 

(61.0, 79.6) 


15.1 

(13.1, 17.1) 


.7 

(-1.5, 2.9) 


Remedial 


58.6 

(50.1,68.7) 


11.2 

(9.4, 13.0) 


3.4* 

(1.5, 5.3) 


Special education 


32.5 

(26.9, 39.3) 


8.8 

(7.2, 10.4) 


2.0* 

(0.3, 3.7) 


Gifted 


57.4 

(46.9, 70.2) 


7.8 

(6.3, 9.3) 


1.4 

(-0.2, 3.0) 


Academic/ College Preparatory 


56.2 

(46.5, 67.8) 


9.0 

(7.3, 10.6) 


2.4* 

(0.7, 4.1) 


Advanced placement /college credit 


- 


1.6 

(0.9, 2.4) 


- 


Honors course 


- 


2.0 

(1.2, 2.8) 


- 


Vocational 


49.2 

(35.2, 68.7) 


2.8 

(1.9, 3.8) 


2.4* 

(1.4, 3.3) 


Bilingual 


66.6 

(52.3, 84.8) 


5.4 

(4.1, 6.7) 


1.9* 

(0.6, 3.2) 


None of the above 


- 


2.4 

(1.5, 3.2) 


- 


* Significantly different from zero. 
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Q 34: Write in your estimate of the percentage of students in your 

DESIGNATED CLASS who were at each level of academic ability for 
their age and grade. (Numbers should total 100.) 



Much above the NATIONAL average % 

Somewhat above the NATIONAL average % 

At the NATIONAL average % 

Somewhat below the NATIONAL average % 

Much below the NATIONAL average % 

O Not applicable 



None of the percentage categories in this question displayed low response 
variance. Table 5 shows the index for each category. 

Combining categories gains a slight reduction in the index of inconsistency 
(See Table 6). Combining the two “above national average” categories and the 
two “below national average” categories yields somewhat more reliable data 
than combining the three middle categories. 

We computed the index using only cases whose percentages summed to 100 in 
both the original survey and the reinterview. About ten percent of the 667 cases 
answering both interviews reported percentages which did not sum to 100 in one 
of the interviews. Comparing the original and reinterview means, medians, and 
standard deviations of these categories provides no evidence that the reinterview 
fails to replicate the original survey. 
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Table 5. 

Q 34: Write in your estimate of the percentage of students in your 

DESIGNATED CLASS who were at each level of academic ability for their 
age and grade. 



n = 601 


Survey 


Mean 


Median 


Standard Deviation 


Index 


Much above the 


Original 


13.88 


5 


22.68 


28.3 


National Average 


Reinterview 


14.23 


5 


22.40 


(4.2) 


Somewhat above 


Original 


17.09 


12 


17.84 


50.7 

(4.4) 


the National 
Average 


Reinterview 


17.22 


13 


17.22 


At the National 


Original 


34.15 


31 


24.71 


36.7 


Average 


Reinterview 


36.33 


36 


25.15 


(3.2) 


Somewhat below 


Original 


20.13 


15 


20.16 


53.6 

(4.9) 


the National 
Average 


Reinterview 


19.55 


15 


19.90 


Much below the 


Reinterview 


12.67 


5 


22.08 


28.8 


National Average 


Original 


14.76 


5 


24.03 


(4.2) 
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Table 6. 

Q 34: Write in your estimate of the percentage of students in your DESIGNATED 
CLASS who were at each level of academic ability for their age and grade. 

Index of inconsistency computed using combined categories 



Response Category 


Categories Not 


Extreme Categories 


Middle Categories 




Combined 


Combined 


Combined 


Much above the 


28.3 




28.3 


National Average 


(4.2) 


22.7 


(4.2) ' 


Somewhat above the 


50.7 


(2.4) 




National Average 


(4.4) 






At the National 


36.7 


36.7 


38.7 

(3.9) 


Average 


(3.2) 


(3.2) 


Somewhat below the 


53.6 






National Average 


(4.9) 


19.1 




Much below the 


28.8 


(2.3) 


28.8 


National Average 


(4.2) 




(4.2) 



The next set of questions, 36, 39, 41, 44, and 49, all follow the same format. 
The questions ask how often the teachers use various teaching methods. 
Response categories for each method are presented in one of two frequency 
scales; 



First Scale 
Almost every day 
Once or twice a week 
Once or twice a month 
Once or twice a semester 
Never 



Second Scale 

Always 

Often 

Sometimes 

Rarely 

Never 



Each question refers to a type of teaching method, such as “instructional 
strategies” or “in-class activities.” Questions within the main question ask about 
specific strategies or activities. Each question within the main question is an 
individual unique question and we analyzed them as such. All these questions 
displayed poor reliability. The aggregate indices, covering the five frequency 
categories, were high for every question and most of the individual category 
indices were high too. The only exception was the “Never” category. For about 
half the questions, the “Never” category displayed moderate response variance. 
The detailed tables in Attachment C reveal no patterns to suggest the cause of 
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this poor reliability. We suspect respondents cannot estimate how often they use 
the various teaching methods. 

The results in Table 7 for question 36 are typical for this group of questions. All 
eleven questions displayed high response variance. Question 36 used the first 
frequency scale. 

Q 36: Over the past semester how often did YOU use each of the following 
instructional strategies with your DESIGNATED CLASS? The 
strategy need not have taken the entire class period. 

Mark (X) one response on each line. 

a. Provide instruction to the class as a whole 

b. Facilitate a discussion 

c. Demonstrate a concept using the board or overhead projector 

d. Work with individual students 

e. Demonstrate a concept using a computer videotape, or other 
electronic medium 

f. Lecture 

g. Work with small groups of students 

h. Lead question-and-answer session 

i. Demonstrate a concept using manipulative s, models, other tools or 
objects 

j. Administer a test or quiz for less than a full period 

k. Administer a test or quiz for a full period 

In several questions the “Almost Every Day” categories displayed 
statistically significant NDRs, suggesting that they did not satisfy the 
response error model assumptions. 
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Table 7. 



Q 36: Over the past semester how often did YOU use each of the 
following instructional strategies with your DESIGNATED 
CLASS... 


Question 


Aggregate Index 


a. 


Provide instruction to the class as a 
whole 


58.9 

(50.6, 68.7) 


b. 


Facilitate a discussion 


58.7 

(53.8, 64.2) 


c. 


Demonstrate a concept using the board or 
overhead projector 


54.4 

(49.9, 59.4) 


d. 


Work with individual students 


63.1 

(56.6, 70.5) 


e. 


Demonstrate a concept using a computer 
videotape, or other electronic medium 


66.1 

(62.6, 70.1) 


f. 


Lecture 


59.4 

(55.8, 63.4) 


g- 


Work with small group 


57.7 

(53.4, 62.4) 


h. 


Lead question-and-answer session 


70.5 

(66.1,75.5) 


i. 


Demonstrate a concept using manipula- 
tives, models, other tools or objects 


60.8 

(57.1,64.9) 


j- 


Administer a test or quiz for less than a 
full period 


53.3 

(49.5, 57.6) 


k. 


Administer a test or quiz for a full period 


49.5 

(45.9, 53.5) 
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Q 39: Over the last semester, how often did you emphasize the following with these 
students? 

Mark (X) one response on each line. 

a. Generalizing from patterns or 

b. Analyzing and interpreting information 

c. Organizing, summarizing, or displaying information 

Questions 39a-c used the first frequency scale. All 3 displayed high response variance 
in all 5 categories. Combining the middle three frequency categories did not 
significantly lower response variance. Table 8 shows that the aggregate index remained 
in the high range for all three questions. 



Table 8. 

Q 39: Over the last semester, how often did you emphasize the following with these 
students? 





Aggregate Index 


Aggregate Index 
(combined categories) 


a. Generalizing from patterns or examples 


79.3 


72.7 




(75.3, 83.7) 


(67.8, 78.2) 


b. Analyzing and interpreting information 


71.2 


70.6 




(67.0, 76.0) 


(65.8, 76.0) 


c. Organizing, summarizing, or displaying 


77.4 


62.0 


information 


(73.4,81.9) 


(57.3, 67.2) 
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Q 41: This following is a list of ACTIVITIES TO COMPLETE A T HOME or 

homework you might have assigned your students. Although the list is not 
exhaustive, most activities could be considered variations of those listed below. 
For each activity described below, indicate the frequency with which you 
assigned each over the last semester in your DESIGNATED CLASS. 

Mark “never" for activities you did not assign during the last semester. 

Mark (X) one response on each line. 

a. Write a journal 

b. Prepare a written report 

c. Work on problems for which there is no obvious method solution 

d. Read the textbook or other assigned reading 

e. Apply concepts or principles to different or unfamiliar situation 

f. Read supplementary material 

g. Complete routine exercises or problems from worksheet, workbook, or text 

h. Work on a project, gather data, conduct an experiment 

i. Prepare an oral report 

j. Complete a short writing assignment 

This question used the first scale of frequency categories. Again, all the questions 
displayed high response variance for almost all the frequency categories. Only the 
“Never” category displayed a moderate index (for eight of the ten questions). 

Q 44: How often do you use assessment information for the following purposes in 
your DESIGNATED CLASS? 

Mark (X) one response on each line. 

a. Determining student grades or other formal progress reports. 

b. Providing feedback to students 

c. Diagnosing student learning problems , 

d. Reporting to parents 

e. Assigning students to different programs or tracks 

f. Planning for future lessons 

Questions 44a-f used the second scale of frequency categories. All the questions 
displayed high response variance. 
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Q 49: How often did you use student portfolios in your DESIGNATED CLASS last 
semester or grading period for the following purposes? 

Mark (X) one response on each line. 

a. Training students to reflect upon and/or assess each piece of work 

b. Training students to reflect upon and/or assess their overall progress 

c. Communicating student progress to parents 

d. Determining student grades or other formal progress reports 

e. Planning for future lessons 

f Diagnosing student learning problems 

g. Making informed decisions about student placement 

h. Making informed decisions about student graduation 

i. Providing information for program or school accountability 

Questions 49a-i used the first scale of frequency categories. They displayed high 
response variance for all of the questions. The NDR for the last three questions was 
statistically significant for all five frequency categories, suggesting that the model 
assumptions were not completely met for these questions. 

The reinterview did not perfectly replicate the original survey for this question due to 
two minor errors in both the mail and telephone reinterview questionnaires. We believe 
neither of these errors invalidates the response error results. 

• The original survey used the first scale of frequency questions, but the 
reinterview used the second scale. Both scales have 5 levels and are almost 
equivalent. 

• The reinterview also omitted question 46, which screened out respondents who 
did not use portfolios in their designated class. This omission caused all 
reinterview respondents to be asked this question. Fortunately, the effect of this 
omission is negligible, because original respondents who did not report “using 
portfolios” were excluded from the reinterview tabulations. 
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Q 43: Estimate the amount of time, in minutes, an average student in your class 
spent doing homework or activities you assigned students in your 
DESIGNATED CLASS to complete at home during an average WEEK. 

Mark (X) only one box. 

O 0 minutes 
O 1-30 minutes 
O 31-60 minutes 
O 61-90 minutes 
O 91-120 minutes 
O 121-150 minutes 
O 151-180 minutes 
O 181-210 minutes 
O 21 1-240 minutes 

Question 43, displayed high response variance, with an aggregate index of 67.0 (63.9, 
70.6). All the individual response categories also displayed a high index of 
inconsistency, except for the “0 minutes” category, which had a moderate index of 28.4 
(23.0, 35.1). 

To determine if the high response variance was caused by too-small time categories, we 
combined the intervals into hours (1-60 minutes, 61-120 minutes, 121-180 minutes, and 
181-240 minutes). Combining categories still resulted in high response variance, with 
an aggregate index of 60.5 (56.9, 64.9). All 4 hourly categories still displayed high 
indices of inconsistency. 

This question requires the respondent to perform several cognitive tasks, such as, 

• Assess how much time the assigned homework should require on average. 

• Remember and evaluate the quantity and quality of homework completed. 

• Consider those two factors to estimate how much time students actually spent 
doing the homework. 

This question also may be subject to the effects of opinion or perception, depending on 
how well the students performed just before the respondent completed the interview. 
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Attachment A-1 



August 17, 1998 

MEMORANDUM FOR: Irwin Schreiner 

Demographic Statistical Methods Division (DSMD) 

From: Andrew Zukerberg 

Demographic Surveys Division (DSD) 

Subject: Teacher Follow-up Survey Reinterview Results 



As requested, I have reviewed the report titled “Response Variance in the 1994-95 Teacher 
Follow-up Survey”. The report presents some interesting findings on the TFS as well as 
suggestions for improvement to future TFS forms. My comments follow the order of the 
report. 

This report documents numerous skip pattern problems. These problems appear to be a 
significant contributor to the overall variance in the survey. This is further evidence that the 
Schools and Staffing Survey could benefit from a computer administration, which would 
eliminate the need for skip patterns. 

Education Field Questions 

I agree with your recommendation that the objectives for the ‘main activities’ section be 
clarified. The intent of these questions is difficult to ascertain from the current administration. 

1 believe that a few problems exist in this section. First, the organization is unclear. The 
respondent is asked if they are teaching either full-time, part-time, or as a long-term substitute. 
If the respondent is teaching, we ask if their time is divided equally between two activities in a 
list of seven. We already know that they are teaching. Next, they are asked to report these two 
activities. If they say no, they are asked to report one activity. An alternative would be to ask 
all respondents to list up to two activities that they engaged in last week. This would eliminate 
the skip pattern and could reduce some of the variance caused by respondents answering 
questions that do not apply. Additionally, there are a couple of ambiguities in this item. There 
is no reference period for ‘your time’. Does this mean your time today or over your lifetime? 
Next, the choices in the list do not fit with the form or question. In the first item respondents 
who indicate that they are no longer teaching are told to stop completing the questionnaire. 
However, we provide a ‘retired’ (06) option to respondents in this list. If a respondent is 
retired, they should have skipped out of the questionnaire before arriving at this item. Option 

02 (working in an elementary or secondary school with an assignment other than teaching) 
seems to preclude a teaching principal from indicating that she has both teaching and 
administrative responsibilities. The category ‘caring for family members’ is also vague. Would 
a respondent who teaches and has children mark this item to indicate that they spend evenings 
and weekends with their children? 
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I share your concern over the clarity of the term ‘work week’. This seems like a vague concept 
that is likely to change frequently during the year. An alternative might be to ask respondents 
about the last full week of school. 

Teaching Assignment / Certification Questions 

I think the suggestion to group main assignment questions separately from other assignment 
areas warrants further exploration. I wonder if part of respondent’s difficulty is related to the 
terms ‘main assignment field’ and ‘other assignment field’. These questions could be worded 
‘the subject you teach the most classes in this semester’. This could help clarify the question 
and eliminate the need for a definition. 

Item 8a; This item is complex in that it asks respondents to think first if they are certified. 

Next, if their certification is in their main assignment field. Finally, they must determine if that 
certification is in this state. It is possible that respondents are not absorbing all three issues. 

For example, a respondent may miss the qualifier “in your main assignment field” when 
initially responding to the questionnaire. They may pick up this reference in the reinterview 
causing them to answer differently. The reference to ‘this state’ could be confusing to the 
people it applies to. If a respondent lives in one state but teaches in another, does this state 
refer to the state where the school is located or they state where they live? 

Item 8b: In cognitive interviews some respondents have reported that they do not know what 
type of certification they have in their main assignment field. This seemed especially true for 
respondents who have been teaching for a long time. This lack of knowledge could account for 
some of the response variance. 

Teaching Methods Questions 

Item 33: I believe the problem of vague terms applies to almost all of the categories. For 
example, to mark bilingual, should the entire class be bilingual or just a few students. 

Item 34: These items ask respondents to compare their students with the national average. 
Although we ask for exact percents, this question is analogous to an opinion question. 
Respondents must estimate what they think the national average is and then think about how 
their class compares. Because this is an estimate, it is not surprising that a respondent’s answer 
shifts between the original interview and the reinterview. 

In cognitive interviews conducted on a series similar to questions 36 - 49, respondents 
indicated that they had a number of difficulties with this section. First, we have observed many 
respondents ignoring the instruction to select a designated class. These respondents answer the 
series of questions across all of the classes they teach. Respondents have a difficult time with 
the reference period. 
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Although we ask about last semester, they tend to indicate that they are answering for their 
current teaching practice. Responses to questions in this series may vary according to the 
content of the lesson. 

For example, a math teacher might use small groups when teaching addition, but lecture when 
teaching subtraction. This could make it difficult for respondents to provide a reliable estimate 
of how often they perform these activities in their classroom. 

CC: Steve Tourkin (DSD) 

Patrick Healy 
Sharon Fondelier 
ESSCB Chron File 
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Response Variance Formulas 



• Original Percentage — the percentage of original responses in a specific answer 
category. The formula is: 

P„ = [{ a+c )/n]x 100 

• Reinterview Percentage — the percentage of reinterview responses in a specific answer 
category. The formula is: 

P,= [{ a+b )/n]x 100 

• Net Difference Rate (NDR) — the difference between the original percent in a specific 
answer category and the reinterview percent in that category. The net difference rate 
measures the net effect of responses changing into and out of that category. The 
formula is: 

NDR = P„-Pr 

= [[(a+c)-(fl+Z;)]/n] X 100 
= [( c-b )/n]x 100 

• Gross Difference Rate (GDR) — the percentage of the responses which change into or 
out of a specific answer category. The formula is: 

GDR = [( Z; + c ) / n ] X 100 

• Simple Response Variance — the average variance of responses from the same units to 
the same question over repeated interviews. The simple response variance equals half 
of the GDR (expressed as a proportion). The formula is: 

SRV={b+c )/ 2n 

• Index of Inconsistency — the ratio (scaled as a percentage) of simple response variance 
to the total population variance for a characteristic. The index represents the proportion 
of the total population variance for a characteristic caused by simple response variance. 

For categorical data, when P = P^ = P^, the formula is: 

Index = [SRV!P{ 1 - P )] x 100 = [[( b-^c )Hn\ / P( 1 - P )] x 100 



where the total population variance for the characteristic is P(l-P). 



Attachment B-2 



For quantitative data, the index equals I - R, where R is the reliability coefficient, a measure of 
reliability used in test theory. R is equal to the conelation between original interview and 
reinterview values, as follows: 



, ^ Cov(y,,,y,2) 

^Var(y.,)Var(y„) 



• Aggregate GDR — the percentage of people who change their answers to a question. 

gor,„=i-E'“„ 

i 

where f*„ = proportion of respondents in category i in both the original and reinterview. 

• Aggregate Index of Inconsistency — an average of indices of inconsistency across all 
categories of the question, weighted by the proportion of cases in each category. 

i-Ep 

Index^a = p 

1-Vp. p 



where P ■ = the proportion of respondents in category / in the original interview 
F, = the proportion of respondents in category / in the reinterview. 
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Is your time EQUALLY DIVIDED between two of the above listed activities? 
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Table 3a Do you spend time on any other activity from the list above? 
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Table 7b(l) Do you teach classes in OTHER fields at this school? 
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Too few cases to produce a reliable estimate 
Estimate is statistically different from zero 
Estimate is undefined 
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Too few cases to produce a reliable estimate 
Estimate is statistically different from zero 
Estimate is undefined 
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Too few cases to produce a reliable estimate 
Estimate is statistically different from zero 
Estimate is undefined 
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Too few cases to produce a reliable estimaite 
Estimate is statistically different from zero 
Estimate is undefined 
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Too few cases to produce a reliable estimate 
Estimate is statistically different from zero 
Estimate is undefined 
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Too Few Cases to Produce a Reliable Estimate 
Estimate is Statistically Different form Zero 
Estimate is Undefined 



VD 



1 

u 


7 






1 




in 


1 










u 






1 






1 








C 


a 






1 






1 








<L> 


S: 






1 






1 
















1 




■ 


1 








C 


0 (D 












1 




1 — . 


1 rH 


jz 


0> 






CO f iH 






1 




1 ^ 




u 


Ai »0 


X 




^ 1 m 


1 in 




1 

» 


as 


1 in 


76 




O 0) 


(P 




in <D 1 ro 






1 




1 in as 


1 in 




3 > 


T3 




. 1 


1 o> 




1 CN 




1 


1 » ' ■» 




(1) ID 


d 




0 i • 


1 in 


r- 




- 


1 ro » 


1 0 0 




E 


H 




00 1 00 0 






1 CN 


00 


1 00 ^ 


1 VD • 




O d 






• 1 <N • 




o^ 


1 






1 00 




4 :: ID 






VD \ ro 




04 


1 


m 


1 ro 


1 ^ 










in 1 oj 






1 


VD 


1 






d» 0» 






w 1 w 






1 






1 ^ 




d d 






1 




a\ 


1 




1 






•H *H 




[ 


1 


• 




1 




. ^ 


^ 




0 V4 
0 




1 


1 


’ T 


’VD 

CN 


1 










TJ 






1 






1 










4J 






1 




w 


1 










d 0) 






1 






1 










Q> E 






1 




_• 


1 










a 0 






1 






1 




1 






CO 






in 1 






1 




1 as 


1 




CO 






• 1 ^ 






1 


• 








CO -U 






in 1 <T\ 






1 


a\ 


1 


1 




ID ID 






sji 1 






1 


CN 


1 iH 






iH 


Q 




1 GO 






1 






1 




U 0) 


0 




. •» 1 CN 


1 'SJ< 




1 0 


«. 


1 0 - - 






AJ 






c>a <J\ 1 • ' 








VD 


1 - 00 


1 rH - 




U Q> 






^ • 1 r* 00 


1 a\ 




1 




1 in • 


1 • as 




d »H 






crv 1 


1 eg 


ro 


1 CN 




1 iH ro 


1 VD • 




0 a 






ro 1 in 






1 


CN 


1 iH 


1 ^ 




>1 E 






— 1 — 






1 


• — • 


1 


1 ^ 




0 






1 




in 


1 










d u 




j 


1 


• 




1 






1 




•H 

0 




1 


1 


1 


CN 

1 


1 
















1 






1 










d 






1 






1 










0) CO 






1 Hi 






1 




1 Hi 


1 Hi 




'O CO 






1 




— 


1 

1 










S3 






1 10 






1 




1 in 


1 rH 




CO 0 












1 


00 




1 * 










1 0 






1 




1 ^ 


1 VD 0 




(P Q 


o: 




1 1 






1 


ro 


1 ro 


1 • 1 




DtU 


Q 




1 og 




0 


1 00 




1 ■ >- 


1 rH 




ID Eh 








1 ID 






«■ 


» CN 0 


1 1 




M < 






\ og 


1 • 


O' 


1 0 


1 — 1 


1 • 


1 0 




p S 






1 1 


1 0 




1 




1 0 






> 0 






1 ro 






1 


CN 


1 


1 ro 




ID H 






1 1 






1 


1 




1 1 




CO 






1 




ro 


1 






1 




d u 






1 




* 


1 










ID Q 




[ 


1 


1 


1 — 1 


1 




I 


I 




- M 




T 


1 


T 




1 




T 


t 




CO d 






\ 






1 










Q> 0 






1 






1 










-U >1 






1 






1 










d 






\ 






1 










d d 






\ 






1 










•H *H 


(P 




1 0 






1 


>-«• 


1 


1 




E 


Di 




1 • 






1 


a\ 


1 0 






CO 


ID 




1 00 






1 in 






1 in 




d -u 


^ H 




1 iH 






1 • 


VD 


1 iH 






•H d 


d p:: 




1 


1 iH 




1 ^ 


CN 


1 1 — 1 


1 




Q) 


(P 




1 <T\ - 






1 CN 




1 ^ 


1 rH 




'T3 


u d 




1 • a\ 


1 ^ 




1 


«. 


1 • 


1 • - 




Q> d 


Jh *H 




\ in • 


1 ^ 




1 


0 


1 as 


I VD 00 




E ^ 


(P 




\ iH ro 






1 


• 








•H CO 


a 




1 iH 






1 


CN 


1 


1 ^ 




4J 






\ 






1 




1 


1 




T? 






1 






1 










4-1 Q> 




[ 


1 






1 




^ 






0 d 

DI 






\ 






1 










AJ -H 






1 




1 — 1 


1 










d CO 






I 






1 










d CO 






1 




in 


1 










0 ID 






\ 

1 




ID 


1 

1 










§ d 






1 






1 










0 






1 






1 










Q> >1 






1 






1 
















1 






1 










^ CO 






1 






1 










(P 












1 










(P -H 






1 






1 










AJ 4J 






1 






1 










ID *H 






1 






1 










E > 






1 






I 










■H *H 






1 






1 










U AJ 






^ 1 






1 










CO U 






ro 1 






1 










a ID 






00 1 

1 






1 

1 
















II 1 






1 
















d 1 
1 






1 

1 
















1 






1 

1 










'O 






• 1 






1 




i CO 


1 CO 




(P 






• 1 






1 CO 




1 (P 


1 (P 




CO 






• 1 






1 (P 




1 4J 


1 4J 




a 






• I 


1 CO 




1 4J 




1 d 


1 d 




ID 






• 1 


1 0) 




1 d 




1 d 


1 d 




iH 






• 1 


i 4J 




1 d 




1 'H 


1 *H 




iH 






• 1 CO 


1 d 




1 ‘H 




1 s 


1 s 




0 






1 0) 


1 d 




» s 










U 






T3 I -n 


1 *H 




1 




1 0 


1 0 










iH 1 d 


1 s 




1 0 




I 00 


1 ^ 










0 1 d 






t CN 




1 rH 


1 CN 




ro 


d 




a 1 -H 


1 0 




1 iH 




1 1 


1 I 






0 




1 1 S 


1 ID 




1 1 




1 rH 


1 rH 






•H 




> 


1 1 




1 iH 




1 CN 


1 00 




0) 


4J 




1 0 


1 tH 




1 ID 




1 rH 


i rH 




tH 


CO 




1 






1 










X) 


<P 




1 






1 










ID 


d 




1 






1 










tH 


o» 




1 






1 









O 




cn> 

cv 



3 

1 

I 

>- 

OL 

O 

o 

Ui 

la 



o 

Q> U 

4J 0) 

<0 N 

AJ M 
U) O 
U <4^ 



0) 4J 
rH C 
^ 0) 
(0 M 
•H 0) 

iH 

(U 4^ 

a: -H 
Q 

(0 

>1 



Q) iH 
U rH 

D <0 
T3 U TD 
O -H 0) 
M C 
CU W -H 
•H 4-1 
O 4J <u 
4J (0 T3 
■H C 
m w D 
0) 

CO (0 U) 



u 

0 ) 0 ) 
S 4J 4J 
0 ) (0 (0 
[K E g 
•H ’H 



O U 4J 
O CO CO 

^ mta 



cvi 

CVl 



I ■*« 4^ 



Table 44a How often do you use assessment information for the following purposes in your DESIGNATED CLASS? 
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Too Few Cases to Produce a Reliable Estimate 
Estimate is Statistically Different form Zero 
Estimate is Undefined 
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Too Few Cases to Produce a Reliable Estimate 
Estimate is Statistically Different form Zero 
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Table 44d How often do you use assessment information for the following purposes in your DESIGNATED CLASS? 
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Table 44e How often do you use assessment information for the following purposes in your DESIGNATED CLASS? 
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Table 49a How often did you use student portfolios in your DESIGNATED CLASS last semester or grading period for the following? 
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-Too few cases to produce a reliable estimate 

* Estimate is statistically different from zero 

# Estimate is undefined 
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Too few cases to produce a reliable estimate 
Estimate is statistically different from zero 
Estimate is undefined 
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Attachment D-1 



Conversion Table for TFS-3 Interview Form Question Numbers and TFS-3(R) 
Reinterview Form Question Numbers 

Note: Attachment E omits pages 2, 10, & 1 1 of the questionnaire, because they were for notes 
only. 



TFS-3 Question Number 


TFS-3(R) Question Number 


la-b 


la-b 


2 


2 


3a-b 


3a-b 


7a-b 


4a-b 


8a-b 


5a-b 


9a-b 


6a-b 


31 


7 


33 


8 


34 


9 


36a-k 


lOa-k 


39a-c 


1 la-c 


41a-j 


12a-j 


43 


13 


44a-f 


14a-f < 


49a-i 


15a-i 
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■M 



as. OErMTHCNr Of OMCHCE 
MmMtQPtiif ccmuv 

«CT«iO «• CQUfiCm« AQOUr faoK 

<is. Off Aimimr of coucaikm 

NAT!QNM CiMTfll rOHEDUCATnMSrATTSTKS 



TEACHER FOLLOW-UP SURVEY 
Reinterview Questionnaire 
for Current Teachere 



1994-1995 



Whan YOU rvsponded to the Tmwher FollowMip Survey 
a abort time ago, were yoo atiil toaehing - full time, 
part-thne, or aa a long>farm aubatituta- in grades K 
.througti 127 



NOTICE - TAN fopoa N auifwyivto Oy i«ag c20 COde 

Vomt onfM^ni be Icept sUicUy cMWentiir^ 

Rwults from this survey wiH «ope«r m mmm mf pr 
lUttttkal lorm only, loihat cannc K bm 

idemfRedL 




lureeuettberoMM 

StWjallc^ 

1201 East lOtfi StNsi 
Jeffereomilla. M 47132-6001 



(Please correct any error in name, 
address, and ZIP Code) 



on] iD YesN» PLEASE COHTINUE with this survey. 



I □ No ^ RETURN; THIS FORM to the Bureau 
of the Census in the enciosed 
envehpe. 



Dear Teacher; 

Thank you for your recent partictpation in the 
Teacher foilo«y*up Survey. As part of conducting a 
survey, we like to evaluate the quality of our data. 

We want to measure how effecnva our questions and 
data collection procedures are at obtaining reliable 
responses. One way to test our questions and 
procedures is to contact original respondents and 
reask selected items from the teacher follow-up 
survey questionnaire you recently mailed us. 

Please complete this reinterview questionnaire with 
information about your teaching at the school named 
on the address label at the righL Return this 
questionnaire to the Bureau of the Census in the 
endosad preaddressed envelope. Please return it 
within 2 weeks. 

If you have questions, please call the Bureau of the 
Census toll free at 1-B00-221-1204. 
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Attachment E-2 




Attachment E-3 









SECnOfif CON1WIUED 




WnidcfBwtw 
Gwwvt- aNmi 



TEACHtNO ASSICmfA^^raaJ9^:(^^ FORI 

■ ■• 57 



Cntw-'Mn^m^ 



111 



Enttryaurt 

inipmMnr A^iMi 6am, mark box 1. and antartha coda frirrfH nfiTiiir (Tn/ff in ijtiiM<lhiii «ft 



Codt 



Main naionniaiit field 



»« 1 



b. Do you W e rt ciaiMwe fii OTHER flelde itt tto artioolT 

J”) r ~ .lOYiW - ■ • . ■ ' s' 



I OOtoliowRe. 1 

Uaa Ifiei i w f dirwiw^ SaU codas ttatadabo^t-ifi-t^' 



[ 

^«Qe 4 

1 



Coda' 



KMaiTf«.sll<ll IVW 



o 

ERIC 
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Attachment E-4 



SECTION I - MAiw Acnyrry ^ contiwiih>. 



te. Do 



«1«i 



Md 7 



MAM ibiwcMng 



I— laY®* 
-jDNo 









Mmrk PO ohhtdn& .biitL ' 



'‘t; 



iV*-l cilIrtatBiBr 






to pmofw «^HAVCC0MK£TED 



wtM ii« sOttp^^ 

certification twao iiim*'i'-^ 

•UTmpomyqftjil^^ additkKitl.CQlW 



to pnrow 

0 f« 0 Ut«r eertineatiOAprof^pnin 

\ ■ ■ 









oOKo|. aocMnii^ - 



;..vS;j 



^□No 




aUl't^C 



what the atata oallaan 
iDPrbviaional 



i:S-SE 

;i b. What type at opffleap^i^i^ thia fiaid? 

■■■.'■■■■ Malt PQ only on» box, 

iDAtlvanoedpwf i a ai o ^ «»t > f ^ 
lORepular or atmdaiti MM oini^^ 

tDTha oertineM offartKl to peraona who HAVE CQI^^ 

oertiflcation progia i w^,;j,%:g,j^^ 

, . ..f ,. f rtap ftidjp^ tn 

c at Cfi c ati on prapfam^O’. y;.-;®. ,v " 

itmid sflw aB 

of a probadortary parU^ 

•bme addftionat edBOoeamiriM 
. , eanba.oKpo(ftsyi:/y>.. 

riQuiaf cartifieatfon :pngrMn In ontor to 



what the awo eallf #o 
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Attachment E-5 



SECnOM U TEACHING RflETHODS 



This Ilia f»acMii9 tlralit9l«Sv inst^ prsciicasr Ofganization^ lachrwiiiM you 

iMin timhing* Th« ititoitm U onyoa provm to <toscnb« •ducsikmal guperionces 

mr^lif4P!^Um iwiowal mivay of 

dmforWlitettV^ had pfinuihjf ir<^^ 1—1 B n m m mtm r or ^fnMng parfed When 

IbOcM^ qum^^ if you Wim mponsMi fpr a iftiglo (such ss 

sh tiPchirm^^ of th^ — tlw dst^iiitiMsd cH—s.' tf you —ris r— 

fec’muN^'bf — a pMPot #f«S: ppa^l odiicatlibo tnctiar ihlghf hsvo 

find oir gi^oup of . die d^ (not homateomK Think of diio as che 



•■7-V.-:-:/;;:-VT^ 














9? BioJowi^. 
5S Chsmtery 




39 Philosoptiy ’ 

40 Phystol sduesttoo, ^?lih 

43 Rasdino"’- 

44 ftsBgloivvf ;•: 

47 il 

Unclu(£«o 



■If ocher scisoce 



l-tecluiicsl edueatfon 
09 Aecoundog 
09 AgrkciAure 

14 Busimta, maricefliia = 

27 HesHh oooipscions 
30 Industrial arts : 

48 Trade and mduBtry : 

90 Technical;;: 

S3 Other yocaCional-iBchnics) 
education/ 



97 ^Special educatlo^gefieral 

98 ': Mentally re ta i 't^ ' ' . ^ 

70 Spe^Kh/tan^i^ 

' 71 - and haid^o^^ 

72 v;stiaily.hafHli^^ 

73 Orih'opedic^^ Im— iin^ ; 

. 74' MiMty he«kifcapped;';:v^.''>/ 
79 Severely handicapped 
79 Specific learn^:c^ 

77 Other, spedal edui^lon ' : 



7. Wlfiet wri— Hie e oh jaot matter of your DCSIQWATED CtASS lest 
Record tte hep dlli^ciodh from the /i^alwe^^ 



semester or 



Cede 



Mairk as^mnem ftsld: 
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^ :'8ECTlMyfiAiCllim ^ ■ ■, 


8. yaw reWWiOTEP CLASS? ' 'V 

twi- Iwt TDAdyanMd^phoamotkfcbl^ 

«»1 tOttawgiiiMOitt . I htI idHaQOisoouiM^:/:.'-^ 

- :>gy^ 

|at*i- wPaWiiBWl ■■ 

, A 


. 0* p^fMMrtfiDs of otudotito : W ^yoiif /DCtt 

. . fPf. inyr,<pp jWW 


qiuirEbi^ctA^ wW '■ . 

ijMbiijiy:-a>;o^ iajlalt lOOjI 




A'- : ■-' 


■ -j-v;v ■ ■ ' 


10. 0»wf ftijpiiit wmiitiltlioiii; eHi dlit YOU a—icli »f thi! 
. fnliiiiiwOliwtwiiWtcwiti^ wMi yowDeSIOflATEDii 


iMmM 

'r-aiiV'ry 


^idi^'bt: 
' ' VMWlt ' 


Oneaor 

twtoaa: 

month; 

■V' « 


OnoaoC'i 

-Mifaaa:. 


iS 


. • ■ ii;'Piib[ii|j|$j||^^ mi • <»o» 






^^'adA 






■■ ■■- ■ :"S. 

Ik Fhtfii^'-«Ota«^nian ' ■ 


^Sr 

y;:id 


%aO"‘ 


»d 


:"*0A 


■'W'- 


■ ■■■ . ■<■ 

'. . c.Oam6hiim|H|iitbaoip(i«ighpihtli^ ' ". . 


mim 


»□ 


id 


'adi-'. 




■• ■ ■ ■■'■ V:,,0-V. 

d.'Wailii*ttiifRtfi^^ 


2tlx 


■■' *0 ■ 


id 


■■' «d"'. 




** lOing a sompular. vktwt^M, gr ottwr 

altcOoniB flMdton ' . - 


»l];y 


; •□ 


ad 


ad 


■■'^"lO'v 


: ^vv;,..:.' ■: ^ 


s*k^ 

y£^;iC] 


tO 


ad . 


aD 


A ‘iO '■ 


9» VHkk iiN|i'0HMl'0reii|09 of itodoRti 
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aO 


■'.d'" 


■aO- 




*J0J * 


f lO ^ 


ad 


■;•; 40t 


H'' lO.;.- 


i OaawwO^^^ models, .' '..y::.'. 

ol^0r looli'oidijicfc . ' -V 


*Jll 


'kiD 


ad 


■•''idA 


■1^3' 


i. AOminMtilll Mil or niz tor Ibbb ttw a M pe^ 




■ fd 


ad 


■:;d;- 




to AdreMator a Mat or a quiz tor a lult^enod 

^■1 *0 ^ t 4MM 


jaj?.-' 

«□ 


»d 


ad 


*d 


u iD 




i?o 
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SECnOW n - TEACtaWG METHOOS -• CO^NUED ; ^ 



It. Owiltm Ut satw— far, liow oft^w yiHi ; 

ttta foHoiifiiig with ttimm tHidanti? 

M§tk (Xk one response on 

«. 0 wwafc*i 9 (ram pMamt oriawplw 


>Atinoat 
-•:ivofy. . 


Oncoor 

tvmoo 

■ 

•■'■.iwi'- 


Onco or 
twice a 
month 

itl 


Once or 
twicea 
aemestar 


Never 

m 




aO ■ ^ 


sO 


40 


»□ 


b. Ai»Wngandinieipr«liiHj Woimb^ 


■asot 

m^o 


)q ■ 


>□ 


40 


»o 


Orfliitring. sunma/atng. or <8iptey(ng| teiswiMaion^EW^^ 


ml 

□ 


■ '^aO;': 


*□ 


«□ 


sO 


fiM foOiMvi^ II ftit of ACtlVniCSTDCOiil!tETC 
HOME or homawofit ypti tniaM hw# <<t>gti#dVo» 
sta«|Mts» Ahhoooli tM Om fr 

ootivMvv. CO 0 M bocoiwMOrwi vsfMtiofio olftlMoo Miitoci . 

fi«q(iMft^.'with iiilikil YOU iiifBiwd aach owacttiiw !. 

■ \ ■■mtiilwr^:ve»DgSK»^^ 

..■ Martir .; /.--7 • 

■ a.WMeaioutnalantoy ' . ' 7;:': 


r’ftiC'';;'''' ■ 




3D 


.□ 


aO 


' V 7b. pfip«foa wriitoo wport ^ ‘ 


agU^:' 


■■■fd-' 


sO 


40 


aO 


e. Wofkon probiBfrai for whibti Ihwiit It w ol soMlon 


ssl,>, 


7' -io : 


»□ 


4D 


aO 


;.'' d. ftoMttfwMxbook'ordttiBr’uaiiOfWdiMi^^ 


»»j 

*□ 


•■ aO' 


>□ 


40 
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